Enhanced plastic recycling using RGB+depth fusion with massFaster and massMask R-CNN

© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksThe rapid increase in waste generation from electrical and electronic equipment (WEEE) has created the need for more advanced sensor-based systems to sort this complex type of waste. Therefore, this study proposes a method for object detection, instance segmentation, and mass estimation of plastics and contaminants using the fusion of RGB and depth (D) images. The methodology is based on the Faster and Mask R-CNN with an extra head for the mass estimation. In addition, a pre-processing method to enhance the depth image (ED) is proposed. To evaluate the data fusion and pre-processing method, two data sets of plastics and impurities were created containing images with and without overlapping samples. The first data set contains 174 RGB images and depth (D) maps of 3146 samples, excluding their mass value, while the second data set contains 42 RGB and D images of 766 pieces together with their mass. The first and second data sets were used to evaluate the performance of Mask and Faster R-CNN. Further, the second data set was used to evaluate the network’s performance with the additional head for mass estimation.The proposed method achieved 0.75 R 2 , 1.39 RMSE, and 0.81 MAE with an IoU greater than 50% using the network Resnet50_FPN_RGBED. Hence, it can be concluded that the presented method can distinguish plastics from other materials with reasonable accuracy. Furthermore, the mass of each detected particle can be estimated individually, which is of great relevance for the recycling sector. Knowing the mass distribution and the percentage of contaminants in a waste stream of mixed plastics can be valuable for adjusting the parameters of upstream and downstream sorting processes.Peer ReviewedPostprint (author's final draft

Simultaneous mass estimation and class classification of scrap metals using deep learning

© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other worksWhile deep learning has helped improve the performance of classification, object detection, and segmentation in recycling, its potential for mass prediction has not yet been explored. Therefore, this study proposes a system for mass prediction with and without feature extraction and selection, including principal component analysis (PCA). These feature extraction methods are evaluated on a combined Cast (C), Wrought (W) and Stainless Steel (SS) image dataset using state-of-the-art machine learning and deep learning algorithms for mass prediction. After that, the best mass prediction framework is combined with a DenseNet classifier, resulting in multiple outputs that perform both object classification and object mass prediction. The proposed architecture consists of a DenseNet neural network for classification and a backpropagation neural network (BPNN) for mass prediction, which uses up to 24 features extracted from depth images. The proposed method obtained 0.82 R2, 0.2 RMSE, and 0.28 MAE for the regression for mass prediction with a classification performance of 95% for the C&W test dataset using the DenseNet+BPNN+PCA model. The DenseNet+BPNN+None model without the selected feature (None) used for the CW&SS test data had a lower performance for both classification of 80% and the regression (0.71 R2, 0.31 RMSE, and 0.32 MAE). The presented method has the potential to improve the monitoring of the mass composition of waste streams and to optimize robotic and pneumatic sorting systems by providing a better understanding of the physical properties of the objects being sorted.Peer ReviewedPostprint (author's final draft

Detection and recognition of batteries on X-Ray images of waste electrical and electronic equipment using deep learning

The trend of increased use of lithium-ion batteries, challenges the cost-effectiveness and safety of manual battery separation during the end-of-life treatment of Waste Electric and Electronic Equipment (WEEE). Therefore, the need for novel techniques to separate and sort batteries from WEEE is increasingly important. For this reason, the presented research investigates the potential to facilitate the development of novel techniques for battery extraction and sorting by examining the technical feasibility of predicting the presence, location, and type of batteries inside electronic devices with a deep learning object detection network using X-Ray images of the internal structure of WEEE. To determine the required X-ray imaging parameters, 532 electronic devices were arbitrarily collected from a recycling facility. From each product, two X-Ray Transmission (XRT) images were captured at two different X-Ray source configurations. Results obtained with the limited dataset are promising, demonstrating a 91% true positive rate and only a 6% false positive rate for classifying battery-containing devices. Moreover, a precision of 89% and a recall of 81% are demonstrated for battery detection, and an average precision of 85% and an average recall of 76% are demonstrated to distinguish amongst the following six battery technologies: cylindrical nickel-metal hydride or nickel-cadmium, cylindrical alkaline, cylindrical zinc-carbon, cylindrical lithium-ion, pouch lithium-ion, and button cell batteries. These results demonstrate the potential of using deep learning object detection on XRT-generated images for both automated battery extraction and sorting, regardless of the condition or shape of the products.status: publishe

You Only Demanufacture Once (YODO): WEEE retrieval using unsupervised learning

Recent developments in robotic demanufacturing raise the potential to increase the cost-efficiency of recycling and recovering resources from Waste of Electrical and Electronic Equipment (WEEE). However, the industrial adoption of robotic demanufacturing for mixed WEEE streams requires tailored instructions for every product model. Considering the large variation in product models, it is not expected to be feasible in the coming decade to rely only on computer vision technologies to define the tailored instructions required for robust and time-efficient robotic demanufacturing. Therefore, the presented research developed a generic retrieval system named You Only Demanufacture Once (YODO) based on content-based image retrieval (CBIR) to identify the product model and retrieve product model-specific demanufacturing instructions. The system compares the visual features represented on a color image of the WEEE with a database of known descriptions representing previously imaged WEEE to find a match or to figure out whether the analyzed product model is new to the system. The performance of YODO is evaluated with a case study for laptop model identification, where a large dataset is created including 4089 images of a representative laptop waste stream. The results demonstrate a top-1 retrieval mean average precision (mAP) of 93.75%. After running YODO on 3600 laptops, the system learned 1079 unique product models, and the presented results show an 85% chance that the next laptop presented to the system is already registered in the database, allowing the retrieval of relevant information for robotic demanufacturing. This corroborates that a fast learning rate can be achieved, allowing a YODO system to support the robotic demanufacturing by making prior product-specific learnings available.Award-winningPostprint (author's final draft

Fostering collaboration between reuse, repair and recycling centers for electric and electronic equipment

When an electronic device fails, treatment procedures and objectives can differ significantly depending on the actor who receives the device. For example, repair facilities generally focus on product reuse and rely on the expertise of employees to select models suitable for repair and subsequent reselling in second-hand stores, while recycling businesses generally focus on bulk processing to recycle raw materials. Even though devices destined for recycling might still qualify for repair or contain valuable/reusable components, there is no method to identify those models quickly in a cost-efficient manner. In addition, tools are lacking that facilitate the registration and retrieval of information on the repairability of specific models or the value of its spare parts. Therefore, an interactive web application has been developed in close collaboration with one of Europe’s largest networks of reuse and repair centers. The developed application can be used when performing triage to determine whether to repair or recycle a specific model. A photograph of the device label is uploaded to an online model identification pipeline. The latter recognizes text on the image with deep learning techniques and compares the text with a database to identify the model, allowing for model-specific information and previous repair experiences to be displayed to the user. Thereafter, novel triage and repair information can be registered and stored for later use. In the presented research, the triage and registration procedure is tested at two repair facilities on 97 washing machines. Learnings from the co-development as well as improvements made throughout the experiment on to the interactive interfaces and forms of the application are presented in this article

Real-time classification of aluminum metal scrap with laser-induced breakdown spectroscopy using deep and other machine learning approaches

In the recycling industry, the use of deep spectral convolutional networks for the purpose of material classification and composition estimation is still limited, despite the great opportunities of these techniques. In this study, the use of Laser-Induced Breakdown Spectroscopy (LIBS), Machine Learning (ML), and Deep Learning (DL) for the three-way sorting of Aluminum (Al) is proposed. Two sample sets of Al scrap are used: one containing 733 pieces for pre-training and validation with a ground truth of X-Ray Fluorescence (XRF), and the second containing 210 pieces for testing for unknown compositions. The proposed method comprises a denoising system combined with a method that extracts 145 features from the raw LIBS spectra. Further, three ML algorithms are assessed to identify the best-performing one to classify unknown pieces of aluminum post-consumer scrap into three commercially interesting output classes. The classified pieces are weighed, melted, and analyzed using spark analysis. Finally, to optimize the best-performing ML system, three state-of-the-art denoising and three feature extraction networks are pre-trained for learning the baseline correction and the proposed feature extraction. Transfer Learning from the six pre-trained networks is applied to create and evaluate 24 end-to-end DL models to classify Al in real-time from >200 spectra simultaneously. The end-to-end DL scheme shows the advantages of learning and denoising the spectra, allowing the transfer of traditional spectral analysis knowledge and the proposed feature extraction into DL, where the network learns from the entire spectrum. The best results for ML and DL were obtained with Random Forest processing one spectrum in 150 ms and BPNN+GHOSTNET(Fine-tuning) processing 200 spectra in 9 ms, which achieved 0.80 Precision, 0.81 Recall, 0.80 F1-score, and 0.80 Precision, 0.79 Recall, 0.79 F1-score, respectively.Peer ReviewedPostprint (author's final draft

Classification of aluminum scrap by laser induced breakdown spectroscopy (LIBS) and RGB + D image fusion using deep learning approaches

Integrating multi-sensor systems to sort and monitor complex waste streams is one of the most recent innovations in the recycling industry. The complementary strengths of Laser-Induced Breakdown Spectroscopy (LIBS) and computer vision systems offer a novel multi-sensor solution for the complex task of sorting aluminum (Al) post-consumer scrap into alloy groups. This study presents two novel methods for fusing RGB and Depth images with LIBS using Deep Learning models. The first method is a single-output model that combines LIBS UNET and two DenseNets in a late fusion framework. The second method is a multiple-output model that uses the structure of the single-output model to enhance learning and avoid overfitting. In particular, the network has two outputs that enable the regularization of the individual sensors. A data set of 773 aluminum scrap pieces was created with two sets of ground truth-values, corresponding to the two envisaged sorting tasks, to train and evaluate the developed models. The first sorting task is separating Cast and Wrought (C&W) aluminum. The second is the division of the post-consumer aluminum scrap into three commercially interesting fractions. The single-output model performs best for separating C&W, with a Precision, Recall, and F1-score of 99%. The multiple-output model performs best for classifying the three selected commercial fractions, with a Precision, Recall, and F-score of 86%, 83%, and 84%, respectively. The presented data fusion method for LIBS and computer vision images encompasses the great potential for sorting post-consumer aluminum scrap. By sorting mixed post-consumer aluminum scrap in alloy groups, more wrought-to-wrought recycling can occur, and quality losses can be mitigated during recycling.Peer ReviewedPostprint (author's final draft